There is a lot of hype about Large Language Models nowadays, but it doesn’t mean that old-school ML approaches now deserve extinction. I doubt that ChatGPT will be helpful if you give it a dataset with hundreds numeric features and ask it to predict a target value.
Neural Networks are usually the best solution in case of unstructured data (for example, texts, images or audio). But, for tabular data, we can still benefit from the good old Random Forest.
The most significant advantages of Random Forest algorithms are the following:
- You only need to do a little data preprocessing.
- It’s rather difficult to screw up with Random Forests. You won’t face overfitting issues if you have enough trees in your ensemble since adding more trees decreases the error.
- It’s easy to interpret results.
That’s why Random Forest could be a good candidate for your first model when starting a new task with tabular data.
In this article, I would like to cover the basics of Random Forests and go through approaches to interpreting model results.
We will learn how to find answers to the following questions:
- What features are important, and which ones are redundant and can be removed?
- How does each feature value affect our target metric?
- What are the factors for each prediction?
- How to estimate the confidence of each prediction?
We will be using the Wine Quality dataset. It shows the relation between wine quality and physicochemical test for the different Portuguese “Vinho Verde” wine variants. We will try to predict wine quality based on wine characteristics.
With decision trees, we don’t need to do a lot of preprocessing:
- We don’t need to create dummy variables since the algorithm can handle it automatically.
- We don’t need to do normalisation or get rid of outliers because only ordering matters. So, Decision Tree based models are resistant to outliers.
However, the scikit-learn realisation of Decision Trees can’t work with categorical variables or Null values. So, we have to handle it ourselves.
Fortunately, there are no missing values in our dataset. df.isna().sum().sum() 0
And we only need to transform the type variable (‘red’ or ‘white’) from string to integer. We can use pandas Categorical transformation for it.
categories = {}
cat_columns = ['type']
for p in cat_columns:
df[p] = pd.Categorical(df[p])
categories[p] = df[p].cat.categories
df[cat_columns] = df[cat_columns].apply(lambda x: x.cat.codes)
print(categories) {'type': Index(['red', 'white'], dtype='object')}
Now, df['type'] equals 0 for red wines and 1 for white vines.
The other crucial part of preprocessing is to split our dataset into train and validation sets. So, we can use a validation set to assess our model’s quality.
import sklearn.model_selection train_df, val_df = sklearn.model_selection.train_test_split(df, test_size=0.2) train_X, train_y = train_df.drop(['quality'], axis = 1), train_df.quality val_X, val_y = val_df.drop(['quality'], axis = 1), val_df.quality print(train_X.shape, val_X.shape) (5197, 12) (1300, 12)
We’ve finished the preprocessing step and are ready to move on to the most exciting part — training models.
Before jumping into the training, let’s spend some time understanding how Random Forests work.
Random Forest is an ensemble of Decision Trees. So, we should start with the elementary building block — Decision Tree.
In our example of predicting wine quality, we will be solving a regression task, so let’s start with it.
Decision Tree: Regression
Let’s fit a default decision tree model.
import sklearn.tree import graphviz model = sklearn.tree.DecisionTreeRegressor(max_depth=3) # I've limited max_depth mostly for visualisation purposes model.fit(train_X, train_y)
One of the most significant advantages of Decision Trees is that we can easily interpret these models — it’s just a set of questions. Let’s visualise it.
dot_data = sklearn.tree.export_graphviz(model, out_file=None,feature_names = train_X.columns,filled = True)
graph = graphviz.Source(dot_data)
# saving tree to png file
png_bytes = graph.pipe(format='png')
with open('decision_tree.png','wb') as f:
f.write(png_bytes)
Graph by author
As you can see, the Decision Tree consists of binary splits. On each node, we are splitting our dataset into 2.
Finally, we calculate predictions for the leaf nodes as an average of all data points in this node.
Side note: Because Decision Tree returns an average of all data points for a leaf node, Decision Trees are pretty bad in extrapolation. So, you need to keep an eye on the feature distributions during training and inference.
Let’s brainstorm how to identify the best split for our dataset. We can start with one variable and define the optimal division for it.
Suppose we have a feature with four unique values: 1, 2, 3 and 4. Then, there are three possible thresholds between them.
Graph by author
We can consequently take each threshold and calculate predicted values for our data as an average value for leaf nodes. Then, we can use these predicted values to get MSE (Mean Square Error) for each threshold. The best split will be the one with the lowest MSE. By default, DecisionTreeRegressor from scikit-learn works similarly and uses MSE as a criterion.
Let’s calculate the best split for sulphates feature manually to understand better how it works.
def get_binary_split_for_param(param, X, y):
uniq_vals = list(sorted(X[param].unique()))
tmp_data = []
for i in range(1, len(uniq_vals)):
threshold = 0.5 * (uniq_vals[i-1] + uniq_vals[i])
# split dataset by threshold
split_left = y[X[param] <= threshold]
split_right = y[X[param] > threshold]
# calculate predicted values for each split
pred_left = split_left.mean()
pred_right = split_right.mean()
num_left = split_left.shape[0]
num_right = split_right.shape[0]
mse_left = ((split_left - pred_left) * (split_left - pred_left)).mean()
mse_right = ((split_right - pred_right) * (split_right - pred_right)).mean()
mse = mse_left * num_left / (num_left + num_right) \\
+ mse_right * num_right / (num_left + num_right)
tmp_data.append({'param': param,
'threshold': threshold,
'mse': mse})
return pd.DataFrame(tmp_data).sort_values('mse')
get_binary_split_for_param('sulphates', train_X, train_y).head(5)
| param | threshold | mse | |---------|------------|-----------| | sulphates | 0.685 | 0.758495 | | sulphates | 0.675 | 0.758794 | | sulphates | 0.705 | 0.759065 | | sulphates | 0.715 | 0.759071 | | sulphates | 0.635 | 0.759495 |
We can see that for sulphates, the best threshold is 0.685 since it gives
Source link
